Illumination systems and methods for computer imagers

ABSTRACT

The present invention can provide solutions to many common imaging problems, such as, for example, unevenly distributed illumination, shadows, white balance adjustment, colored ambient light and high dynamic range imaging. Imaging systems and methods can be provided through a computer (e.g., laptop or desktop) such that the system or method can take advantage of the computer&#39;s processing power to provide functionality that goes beyond typical camera. Such an imaging system may include an imaging device, a camera, a light source and a user interface.

The present application is a continuation of application Ser. No.15/404,951, filed on Jan. 12, 2017, now U.S. Pat. No. 10/104,303, whichis a continuation of application Ser. No. 14/581,894, filed Dec. 23,2014, now U.S. Pat. No. 9,571,745, which is a division of applicationSer. No. 12/006,637, filed Jan. 3, 2008, now U.S. Pat. No. 8,922,672.

BACKGROUND OF THE INVENTION

This relates imaging systems.

Most imaging systems use artificial lights, such as a camera flash, toilluminate a scene for image capture. Systems typically use theseartificial lights to emit the same color and brightness of light when animage is captured. Therefore, traditional imaging systems have limitedcontrol over the light used to artificially illuminate a scene for imagecapture and are incapable of changing the color or brightness of thelight in response to preexisting lighting conditions (i.e., ambientlight).

In other instances, when images are to be captured, there is no choicebut to turn on the room lights, such as during traditionalvideoconferencing. This can present problems, particularly in instanceswhere it is desired to keep room lights off. Thus, there is a need toprovide improved imaging systems and methods.

SUMMARY OF THE INVENTION

The present invention can provide solutions to many common imagingproblems, such as, for example, unevenly distributed illumination,shadows, white balance adjustment, colored ambient light and highdynamic range imaging. In some embodiments, the imaging systems andmethods are provided through a computer (e.g., laptop or desktop) suchthat the system or method can take advantage of the computer'sprocessing power to provide functionality that goes beyond typicalcamera.

In one embodiment, an imaging system can include an imaging device, acamera, a light source and a user interface. The system an analyzecaptured images to determine if light is evenly balanced across theimage and, if not, adjust the intensity of one or more lights to balancethe light so that a higher quality image can then be captured.

In other embodiments, the system can collect data from the surroundingenvironment which can then be analyzed to generate ambient lightparameters. The system can then vary the illumination to the imagetarget (the object or scene that is to be captured) based on the ambientlight parameters. The image would then be captured while the adjustedillumination was activated.

In other embodiments, the system can capture a first image while thescene is illuminated, and then capture a second image when theillumination is either turned off or turned away from the scene. Toproduce the “captured image”, the system can then combine the two imagesto produce the final image, which should be of a higher quality thaneither of the captured images.

In another embodiment, the system collects data regarding the scene. Thescene can be divided into portions and ambient light parameters can begenerated for each portion. For example, the ambient light parametersmay indicate that one or more portions may be dimly reflecting whileother portions may be brightly reflecting. Illumination can then beprovided appropriate to the reflectivity since the illumination would bebased on the ambient light parameter for each portion instead of asingle ambient light analysis.

In yet another embodiment, the scene can still be divided into portions,each of which can be analyzed to produce a separate ambient parameter.The system could then illuminate one portion at a time, and capture theimage of that portion. Once all of the portions have captured, thesystem could then combine the images into a single image.

In another embodiment of the present invention, the system can captureimages of a scene under multiple levels of illumination, and combine theimages into a single image. For example, the system can illuminate thescene with a certain light level appropriate to the most reflective areaof the scene and capture an image of the scene under that illumination.Before the scene changes, for example, due to subject movement, thesystem can illuminate the scene with a different light levelappropriate, for example, to the least reflective area of the scene, andcapture an image of the scene under the different light level. Thesystem also can capture images of the scene under other intermediateillumination levels (in any order). Once all of the brightness levelshave been captured, the system then can combine the images into a singleimage.

In another embodiment, the system can include an image capturing device,such as a digital camera, processing circuitry and one or moreillumination devices. The processing circuitry can be configured toanalyze the ambient light in the scene surrounding an object to be“captured.” This can be done either as a single analysis or as a seriesof portion analyses. The image(s) is then captured; in the case ofportions, the series of sub-images are captured and then combined.

Persons of ordinary skill in the art will appreciate that the variousembodiments described herein can be combined with other describedembodiments or other embodiments without departing from the spirit ofthe present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other advantages the present invention will be apparentupon consideration of the following detailed description, taken inconjunction with accompanying drawings, in which like referencecharacters refer to like parts throughout, and in which;

FIG. 1 is a simplified diagram of an exemplary imaging system inaccordance with an embodiment of the present invention;

FIGS. 2A-2C are illustrations of an exemplary integrated light inaccordance with an embodiment of the present invention;

FIG. 3 is a flowchart of an illustrative method for capturing a digitalimage using one or more lights in accordance with an embodiment of thepresent invention;

FIG. 4 is a flowchart of an illustrative method for capturing a digitalimage using automatic white balancing through illumination in accordancewith an embodiment of the present invention;

FIG. 5A is a flowchart of an illustrative method for capturing a digitalimage without any color from ambient light in accordance with anembodiment of the present invention;

FIG. 5B is a flowchart of an illustrative method for providing a highdynamic range image in accordance with one embodiment of the presentinvention; and

FIG. 6 is a flowchart of an illustrative method for capturing an imageof a scene having high dynamic range in accordance with an embodiment ofthe present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to imaging systems. An imaging system isan electronic system which is able to capture an image. In oneembodiment, imaging systems can include an imaging device and any numberof accessories.

FIG. 1 is an illustration of exemplary imaging system 100 in accordancewith one embodiment of the present invention. In one embodiment, imagingsystem 100 may perform functions in addition to image capture. Forexample, imaging system 100 can be a computer that also performscomplicated processes (e.g., word processing and mathematicalcalculations). In another embodiment, imaging system 100 can be acamera. Imaging system 100 can include imaging device 110, camera 120,light 130 and user interface 140.

Imaging device 110 can coordinate the functions of imaging system 100.Processor 112 can be provided within imaging device 110 or it can beprovided in another component of imaging system 100. Processor 112 can,for example, include a processor, a field-programmable gate array, anapplication-specific integrated circuit, or a combination of individuallogic components. Processor 112 can include circuitry that is optimizedfor processing digital images. Imaging device 110 can include memory andother circuitry (not shown). For example, imaging device 110 can includecircuitry for storing or transmitting captured images.

Camera 120 can include a lens and one or more sensors that generatedigital images. The sensors of camera 120 can be provided on acharge-coupled device (CCD) integrated circuit, for example. Camera 120can include dedicated image processing circuits for converting signalsfrom one or more sensors to a digital format. Camera 120 can alsoinclude circuitry for pre-processing digital images before they aretransmitted to imaging device 110.

Light 130 can generate light to aide in capturing images. Light 130 caninclude one or more types of lighting elements, such as an incandescentbulb, a halogen bulb, a fluorescent bulb, a high-intensity dischargelamp, a light emitting diode (LED), an arc lamp, an electronic xenonflash lamp, a mcroflash or other suitable lighting element. Imagingsystem 100 may be able to control the color and brightness ofillumination from light 130.

In one embodiment, light 130 can include one or more lighting elementsof different colors. The one or more elements can be independentlyadjusted to output light of varying intensity (e.g., brightness). Byadjusting the relative intensity of differently colored lightingelements, the light source can emit a large range of colors. Forexample, if red, green and blue lighting elements are used, the relativeintensity of each element can be adjusted to create a blend of lightthat corresponds to an RGB color. In addition to one or more coloredelements, it may be advantageous to also include a substantially whitelighting element which can efficiently add to the total intensity oflight without significantly affecting the color.

User interface 140 can receive user instructions. For example, userinterface 140 can include individual buttons, a keyboard, a pointingdevice (e.g., a mouse), or a touch-screen display. In one embodiment, auser can employ user interface 140 to control when and how imagingsystem 100 captures images.

User interface 140 can also present information to a user. User,interface 140 can include a display screen, an audio system, or aprojector. User interface 140 can display previously captured images toa user, for example. User interface 140 may also instruct a user on howto configure imaging system 100.

In some embodiment, imaging system 100 may include an ambient lightdetector (not shown). An ambient light detector may be used to measurethe brightness and color of preexisting lighting conditions. Thisinformation may be useful in order to properly control light 130 suchthat a scene is illuminated appropriately when an image is beingcaptured. An ambient light detector can be included in any portion ofsystem 100, such as in imaging device 110, camera 120, light 130 or userinterface 140. Alternatively to or in combination with an ambient lightdetector, a sample image may be captured by camera 120 to measure thepreexisting lighting conditions for the same purpose.

In some embodiments, camera 120, light 130, and user interface 140 canbe separate accessories that work with imaging device 110. Camera 120,light 130, and user interface 140 can connect with each other or imagingdevice 110 through a wired or wireless connection.

Alternatively, any one of imaging device 110, camera 120, light 130, anduser interface 140 can be combined into a multi-function device. Forexample, camera 120 and light 130 can be combined into one device thatcommunicates with imaging device 110. In another example, imaging device110, camera 120, light 130, and user interface 140 can all be combinedinto a single device (e.g., a laptop computer or a digital camera).

In one embodiment of the present invention, light 130 and user interface140 can be combined in the form of a display screen (e.g., CRT screen,plasma screen, or LCD screen). A display screen may be a desirable lightbecause it can generate light that varies in both brightness and color.A display screen can also be a user interface because it can presentinformation to a user. Moreover, a display can also receive instructionsfrom a user if it is a touch-screen.

It is understood that one or more cameras, lights, or user interfacescan be included in an imaging system without deviating from the spiritor scope of the present invention. For example, an imaging system mayinclude multiple lights to illuminate a scene from multiple angles forimage capture.

FIG. 2A is an illustration of imaging system 200 in accordance with oneembodiment of the present invention. Imaging system 200 can includeimaging device 210 (e.g., imaging device 110 of FIG. 1).

In one embodiment, imaging device 210 can be a computer. Imaging device210 can include a processor (e.g., processor 112 of FIG. 1) thatcontrols the functions of imaging device 210 and any external components(e.g. 210 cameras, lights or user interfaces). Imaging device 210 canalso include memory (not shown) for storing configuration data andcaptured images.

Imaging device 210 can include embedded camera 212 and connectors or awireless system for communicating with one or more external cameras.Embedded camera 212 can be embedded within the housing of imaging device210, and any external cameras might include their own housing. Embeddedcamera 212 and any external cameras coupled with device 210 may becomparable to camera 120 of FIG. 1.

Embedded camera 212 and any external cameras can capture images.Embedded camera 212 and any external cameras may be able to convertcaptured images to digital pictures or may transmit a signal to aprocessor in device 210 for conversion. Embedded camera 212 can bepositioned in imaging device 210 so that, for example, it is aimed atthe area where a user's face would typically be when using imagingdevice 210. In some embodiments, embedded camera 212 may be movable suchthat it can be pointed in a particular direction. External cameras maybe positioned and aimed independently of imaging device 210. Processor214 can control when and how camera 212 and any external cameras captureimages.

Imaging device 210 can include embedded light 216 and connectors orwireless systems for communicating with one or more external lights. Forexample, an external light can include one or more lighting elementsmounted on a tri-pod and can be wirelessly connected to an imagingsystem. Embedded light 216 can be incorporated into the housing ofimaging device 210. Embedded light 216 and any external lights may becomparable to light 130 of FIG. 1.

Embedded light 216 and any external lights can provide it during imagecapture. For example, embedded light 216 can be positioned in imagingdevice so that, for example, it is pointed in the same direction ascamera 212. In some embodiments, embedded light 216 may be movable sothat it can be pointed in a particular direction. Processor 214 cancontrol the color and intensity of light emitted by embedded light 216and any external lights.

Embedded light 216 and any external lights can include one or morelighting elements. The lighting elements of embedded light 216 and anyexternal lights can be different colors such that the combination oflight creates a particular color. In one embodiment, embedded light 216and any external lights can include three colored lighting elements andone white lighting element. For example, the lighting elements caninclude a red LED, a green LED, a blue LED and a white LED. The coloredlighting elements can be independently controlled (e.g., by a processorin device 210) to emit light of different relative intensities. Therelative intensities can be selected so that the blend of light from allthree sources creates a desired color. Additionally, the white lightingelement can be illuminated in order to add to the overall intensity oflight generated by light sources.

Embedded light 216 and any external lights may include driver circuitry(not shown). Driver circuitry can, for example, receive an electronicsignal with color and timing information and convert into individualsignals to drive one or more lighting elements. Driver circuitry canensure that each lighting element is illuminated at the designated timewith the designated intensity. Driver circuitry can change the intensityof light from lighting elements by adjusting the signal used to powerthe element. Driver circuitry can, for example, vary the duty cycle,current flow or any other suitable aspect of a signal to control thebrightness of each lighting element. Driver circuitry store electricalenergy. For example, driver circuitry can include one or more batteriesor capacitors for storing energy that allows lighting elements to outputquick bursts of high intensity light even if the power supply's currentis limited.

Imaging device 210 can include display screen 218, buttons 220, andconnectors or a wireless system for communicating with one or moreexternal user interface accessories (e.g., a keyboard or a mouse) (notshown). Display screen 218, buttons 220, and external user interfaceaccessories may be comparable to user interface 140 of FIG. 1. Displayscreen 218, buttons 220, and any external user interface accessoriesmaybe coupled with a processor in device 210. A user can employ buttons220 and/or any external user interface accessories to provide input toimaging system 200. Display screen 218 can present the output of imagingsystem 200 to a user.

Display screen 218 can also function as a light source that providesillumination during image capture. For example, processor 214 caninstruct display screen 218 to output a predetermined amount and/orcolor of light while camera 212 is capturing an image. One example ofthis embodiment is described in greater detail below with respect toFIG. 2C.

It is understood that any external cameras, external lights, or externaluser interface accessories can connect with imaging device 210 throughone or more electrical connections (e.g., USB or Firewire cables) orwireless communications.

Alternatively or in addition to light 216, device 210 may includeembedded lights 230 and 250 in accordance with one embodiment of thepresent invention. FIG. 2B is an illustration of imaging system 200 withembedded lights 230 and 250 deployed. Embedded lights 230 and 250 may becomparable to light 130 of FIG. 1. Embedded lights 230 and 250 canprovide illumination during image capture. A processor in device 210 cancontrol the deployment and operation of embedded lights 230 and 250.Embedded light 250 is similar to embedded light 230, and the followingdescription of the embedded light 230 can also be applied to embeddedlight 250.

Embedded light 230 can include a reflector 232 and one or more lightingelements 234. Reflector 232 can include material that reflects light.Lighting elements 234 can be aimed at the front surface of reflector 232such that any light from lighting elements 234 reflects off of reflector232 in a manner that provides substantially even illumination. Whenfully deployed, reflector 232 can be tilted towards the front of imagingdevice 210 such that light reflected from lighting elements 234 isdirected towards a person using device 210.

Embedded light 230 can include support appendages 236-244. Supportappendages 236-244 can be used to support deploy/stow reflector 232.Support appendages 236-244 can extend radially from a corner of imagingdevice 210. The ends of support appendages 236-244 can be pivotallyengaged with a joint in the corner of imaging device 210. The positionof support appendages 236-244 can be controlled by a motor (not shown)imaging device 210. Support appendages 236-244 can be made of a materialthat is substantially rigid enough to support reflector 232.

Reflector 232 can be coupled with support appendages 236-244 by affixingthe reflector to the appendages using, for example, an adhesive or othersuitable coupling means. For example, appendages 236 and 244 can becoupled to the edges of reflector 232, and appendages 238-242 can becoupled to the rear surface of reflector 232. Appendages 238-242 can becoupled to reflector 232 at evenly spaced intervals.

Embedded light 230 can be stowed such that reflector 232 and supportappendages 236-244 are located within cavity 246. Embedded light 230 canbe stowed by, for example, rotating support apertures 236-244 about apivot point in the corner of device 210. Reflector 232 can be flexibleso that when embedded light 230 is stowed, the reflector can be easilycompressed or folded. Appendage 236 can be shaped to fit the opening ofcavity 246 such that the cavity is cannot be easily seen when embeddedlight 230 is stowed. Imaging device 210 of FIG. 2B can resemble imagingdevice 210 of FIG. 2A when embedded lights 230 and 250 are stowed.

It is to be understood that the above imaging systems are providedsolely for the purposes of illustration and that other imaging systemscan be used without deviating from the spirit and scope of the presentinvention.

In one embodiment of the present invention, an imaging system (e.g.,system 100) may include one or more lights (e.g., light 130) and thesystem may be able to adjust the one or more lights to properlyilluminate a scene.

FIG. 2C is an illustration of another exemplary integrated light inaccordance with an embodiment of the present invention. Imaging system250 can be configured to use a display screen as the light forilluminating a subject during, for example, video-conference. In oneembodiment, imaging system 260 can include embedded camera 262 andconnectors or a wireless system for communicating with one or moreexternal cameras. Imaging system 260 also can include screen 264 onwhich image 266 of the subject can be presented during thevideo-conference. While image 266 can be disposed anywhere on screen264, the image can be placed near camera 262 so that the subject appearsto be looking into the camera as he observes the transmitted image. Theremainder of screen 264 can set to bright white light (e.g., RGB=255,255, 255) to illuminate the subject as brightly as possible. Theremainder of screen 264 also can beset to an illumination level based onthe level of ambient light detected by one or more photodetectors 268.

Based on an analysis of the color balance of image 266, imaging system260 also can adjust the screen color. For example, if the image appearstoo blue, the RGB values may be set to 240, 255, and 255, respectively.Imaging system 260 also can capture multiple images of a subject andadjust the illumination level and color of screen 264 to create, forexample, uniform and color-balanced light for the subject. For example,the imaging system can take images with and without screen illuminationon during the exposure (see, e.g., the discussion corresponding to FIG.5A). When screen illumination is not desired (e.g., to take an imageonly with ambient light), screen 263 can be black. Image 266 also can bedimmed or blackened momentarily during the exposure.

FIG. 3 shows an illustrative flowchart of method 300 for capturing animage in accordance with an embodiment of the present invention. Duringmethod 300, lights may be adjusted by instructing a user to move theirposition or adjusting their relative brightness.

At step 310, a digital image can be captured. The image can be capturedusing a camera (e.g., camera 120). In some embodiments, such as systemswith independently moveable lights, a system may record the position ofthe lights when the image is captured. The position of each light may bedetermined automatically (e.g., based on wireless communications) or maybe provided through a user input.

At step 320, the digital image can be analyzed. The camera used to takethe image or a processor in the imaging device (e.g., processor 112) cananalyze the image. Analyzing an image can determine if any lights areimproperly positioned by detecting shadows or bright spots in the image.In one embodiment, an image can be analyzed to identify any regions ofhigh and low color pixel saturation (e.g., areas with a large number oflight or dark red, green, and/or blue pixels). For example, regions ofthe image with a large number of light pixels may be categorized asbright spots and regions with a large number of dark pixels may becategorized as shadows.

In some embodiments, the entire image may be analyzed as a whole toidentify any groupings of light or dark pixels. In some embodiments, theimage may be broken down into pre-defined regions (e.g., 4 or 16regions) and each region may be analyzed separately to determine if itis a bright spot, a shadow or neutral. As part of the analysis, theimaging system may determine whether or not the light in the image isevenly distributed. For example, the imaging system may use thelocations of the bright spots and/or shadows in the image to determineif the light is not uniform. For example, a large number of bright spotson one side of an image may indicate that a light on that side isunevenly illuminating the image. In some embodiments, the imaging systemmay use the relative locations of the bright spots and shadows. Forexample, if many pairs of bright spots and shadows are identified andthe bright spots are always to one side of the shadows then a light onthat side may be unevenly illuminating the image.

At step 330, method 300 may diverge depending on the analysis performedin step 320. If the imaging system determines that the light in theimage is unevenly distributed, method 300 may proceed with step 340. Atstep 340, the system may adjust one or more lights to correct thedeficiency. For example, the system may instruct a user to repositionone or more lights. The system may provide specific instructions to auser detailing which light to move and where to place it. The system mayinstruct the user through a user interface (e.g., user interface 140).The system may adjust one or more lights automatically through wired orwireless connections.

In some embodiments, the system may adjust the relative brightness ofone or more lights. In embodiments where a display screen is used as alight, the light can be adjusted by illuminating different areas of thedisplay screen. This adjustment of the display screen is comparable tochanging the position of the light. For example, if only one side of thedisplay screen is illuminated during image capture, the resulting imagemay resemble an image that was generated with an external light placedon that side. After the one or more lights have been adjusted, thesystem may proceed with step 310 and capture another image. At thispoint, method 300 would then analyze the new image and make anotherdetermination of whether or not the light is evenly distributed.

If the imaging system determines that the light is evenly distributed,method 300 may end with step 350. At step 350, the system may store theimage in memory. For example, the image may be stored on memory in animaging device (e.g., device 110). The position of the lights, if known,may be stored so that it is appended to or part of the image data. Thisdata may provide useful information about the image at a later time andmay be useful for editing the image.

FIG. 4 shows an illustrative flowchart of method 400 for capturing animage in accordance with one embodiment of the present invention.According to method 400, an imaging system (e.g., system 100) cancollect data from one or more sources and generate ambient lightparameters from the collected data. After generating parameters, thesystem can then illuminate a scene (e.g., using light 130) according theparameters and capture an image during the illumination. Compared totraditional image capturing with standard illumination, method 400 canresult in higher quality images with truer colors.

At step 410, data can be collected from one or more sources. Thecollected data can reflect characteristics of the ambient light in ascene. An imaging system can collect data from one or more types ofsources, including a photodetector (e.g., a CCD array in camera 120 oran ambient light sensor), a user input device (e.g., user interface140), or another imaging system.

For example, data can be collected from the output of one or morephotodetectors. For example, the photodetectors can be provided througha CCD array in a camera (e.g., camera 120). Alternatively, the one ormore photodetectors can be provided through an ambient light sensor(e.g., one or more photodetectors located behind colored filters).Compared to a camera, an ambient light sensor may provide less accuratedata about ambient light but may be less expensive and not require asmuch signal processing or power consumption. In either form, the datacan be previously collected for other purposes (e.g., the last imagetaken by the camera, a previous frame of a video feed or a measurementto adjust the brightness of a display screen) such that collecting thedata merely requires accessing the previous data. For example,collecting the data may be accomplished by accessing a previouslycaptured image. In other embodiments, data can be collected for theprimary purpose of gaining information about ambient light.

Additionally or alternatively, a user can manually input data aboutambient light through a user interface (e.g., user interface 140). Thiscan be advantageous if a user has a separate, more accurate sensor thanthe sensor provided in an imaging system. However, this form of data canalso be generated from less precise user inputs. For example, a user maybe able to indicate what the conditions are when an image is captured(e.g., direct sunlight, shade, fluorescent lighting).

In some embodiments, data can be collected from another imaging system.For example, if an additional imaging system is subject to the samelighting conditions, the two systems can use the same data. Sharing databetween imaging systems can be advantageous if the systems aregenerating images that will later be associated together and, therefore,may need to be illuminated in a similar manner.

In some situations, data may be collected while an imaging system wasproviding illumination. In such a situation, information about theillumination (e.g., color and intensity) can be incorporated into thedata. This information can be used at a later time to compensate for anyeffects that the illumination may have on the collected data. Forexample, when the data is processed the illumination provided by thesystem at the time of data collection may be used to account for theeffect of that illumination.

At step 420, ambient light parameters can be generated from thecollected data. Ambient light parameters can reflect the color andintensity of ambient light in a scene. For example, ambient lightparameters can include a measurement for each of the three primarycolors of light (i.e., red, blue, and green). Ambient light parametersmay include the color temperature of the ambient light.

There are several algorithms that can be used to generate ambient lightparameters. In one embodiment, the color value of all pixels in an imagecan be averaged to determine the color and intensity of ambient light inthe image. In some embodiments, the color value of each pixel may beweighted based on its position in the image. For example, a pixel on theedge of an image might have less weight that a pixel near the center ofthe image in determining the overall ambient light. If an imaging systemprovided illumination when the image was captured, the color of thatillumination can be used to adjust the average color value of all of thepixels.

In another embodiment, an imaging system can automatically identify anobject in an image and assume that it is white. The system can thenaverage the color value of the pixels corresponding to the object todetermine the color and intensity of ambient light when the image wascaptured. In one embodiment, an imaging system may instruct a user tohold up a white object that can be used to determine the color ofambient light in the image. In another embodiment, an imaging system canautomatically identify objects that are typically white (e.g., buttonson a shirt) and can analyze the pixels corresponding to that object. Insome embodiments, a user may be able to provide an input specifying whatportion of an image the system can assume is white. Moreover, componentsof an imaging system (e.g., a mouse or a keyboard) can be colored whitesuch that, if they are shown in an image, these components can be usedas reference objects when determining the color and intensity of ambientlight.

In yet another embodiment, an imaging system can identify neutrallycolored metal objects and can analyze the pixels showing light reflectedoff of those objects to characterize the color and intensity of ambientlight when the image was captured.

For imaging systems that use an ambient light sensor, generating ambientlight parameters may not require as much image processing. For example,the color and intensity of ambient light can be determined by measuringdata from the sensor. In some embodiments, the measured data may beadjusted to compensate for any illumination provided by the system whenthe data was collected.

Other algorithms for determining the color and intensity of ambientlight in an image are well known in the art. Any of these algorithms canbe used without deviating from the spirit or scope of the presentinvention.

Ambient light parameters can be generated by software running on aprocessor. Alternatively, a system can use application specific hardwareto process data and determine ambient light parameters. Such a processoror application specific hardware can be located anywhere in an imagingsystem. For example, application specific hardware can be provided in acamera (e.g., camera 120) or in an imaging device (e.g., device 110).

At step 430, an imaging system can provide illumination based on thegenerated ambient light parameters. An imaging system can use one ormore lights (e.g., light 130) to provide illumination. A light can beincorporated into an imaging device (see, e.g., light 130 or displayscreen 140) or external to an imaging device (e.g., an external light).Illumination can be provided for a period of time that is suitable forimage capture.

The illumination provided by an imaging system can have a particularcolor and/or intensity that is selectively chosen according to one ormore ambient light parameters. An imaging system with multiple lightsources can provide different illumination from each light source.

The illumination provided by an imaging system can be used to balancethe color of a scene. In one embodiment, the illumination car be a colorthat is substantially complementary to the detected ambient light'scolor. For example, if ambient light parameters identify that theambient light is primarily composed of red light, the imaging system canprovide illumination that is primarily composed of green and blue light(or yellow and blue light).

It is to be understood that the color of illumination provided by anillumination system may be limited by the lights, the colors of lightingelements therein or the resolution of control that the system has overthose lighting elements. For example, if an imaging system only has redand blue lighting elements, it may balance a red ambient light byproviding blue illumination.

In addition to being a complementary color, the intensity of theillumination may be proportional to the intensity of the ambient light.In this manner, the imaging system may provide illumination thatneutralizes the color of any ambient light. This can result in an imagewith realistic colors that aren't affected by the color tint of theambient light.

At step 440, an image can be captured while illumination is beingprovided. An image can be captured by a camera (e.g., camera 120),converted to a digital format, and stored in the imaging system (e.g.,on imaging device 110). The captured image may be color balanced becauseof the color and intensity of the illumination that was provided duringimage capture.

Information about the illumination (e.g., its color and intensity) thatis provided when an image is captured may be appended to or part of theimage data. This information may then be used at a later time, such aswhen a user editing the image. For example, a user may decide to removethe effects of the colored illumination and therefore show the effect ofthe ambient light.

FIGS. 5A-5B show illustrative flowcharts of methods for capturing andcombining images in accordance with embodiments of the presentinvention. In methods 500 and 520, illumination of a different colorand/or intensity level can be provided when capturing each image beforethe images are combined. For example, a first image can be capturedusing one level of illumination, a second image can be captured using asecond level of illumination, and both the first and second images canbe combined to form a final image. In some embodiments, the differentlevels of illumination can be used to negate the color of ambient lightin a scene. For example, subtracting a naturally illuminated image(e.g., captured with only ambient light) from a neutrally illuminatedimage (e.g., captured with neutral or white light from the imagingsystem) can minimize the effect of any ambient light.

When capturing a series of images to be combined later, there arecertain qualities that can be advantageous in an imaging system. Forexample, it may be beneficial to use a system that can capture images inrapid succession such that the scene does not substantially change overthe consecutive images. A change in the scene (e.g., a subject moving)create difficulties when combining the images. Additionally, it may beadvantageous to use a system than can adjust illumination quickly sothat the system is prepared to provide proper illumination for eachimage capture in a rapid, consecutive series.

In process 500, an imaging system can turn off the illumination (step502) in order to capture a naturally illuminated image step 504). Instep 506, the imaging system can turn on the illumination. For example,the illumination intensity and color level can be predetermined valuesor can be based on one or more factors discussed herein in accordancewith the present invention (e.g., ambient light, color balance, etc.).The color and intensity of the light may be chosen so that it issufficient to illuminate a scene with no ambient light. For example, thelight's color may be a balanced white and the light's intensity may bebright enough to sufficiently illuminate the scene, assuming the absenceof ambient light. At step 508, the imaging system can capture an imageof the scene with the subject illuminated by the imaging system. Steps502-508 may occur in rapid succession so that the scene does not changesubstantially between the two images. In some embodiments, an imagingsystem may instruct a user (e.g., using user interface 140) to remainstill during image capture.

At step 510, the imaging system can combine the images captured in steps504 and 508. An imaging system may include a processor (e.g., processor112) application specific hardware for combining the images. Theprocessor or application specific hardware that combines the images maybe located anywhere in the imaging system (e.g., in camera 120 orimaging device 110).

In some embodiments, the imaging system may combine the images bysubtracting the image captured in step 504 from the image captured instep 508. In such an embodiment, each image may have been captured thesame amount of ambient light so that the subtracting one image from theother minimizes the effect of that light. Accordingly, the combinedimage may not include any effects of the ambient light when the imageswere captured. Such a process may be useful when, for example, theambient light is tinted an undesirable color. For example, if theambient light were substantially red, method 500 can be used to minimizethe effect of the red light on the final image.

The combined image may be stored on the imaging system for later use.Data relating to the illumination and/or the time difference between theimages may be stored with or appended to the image data. Such data maybe useful when editing or analyzing the image.

FIG. 5B is a flowchart of an illustrative method for providing a highdynamic range image in accordance with one embodiment of the presentinvention. Currently, techniques are available to acquire a high dynamicrange image by combining and tone-mapping multiple images of a scenecaptured by using various exposures. However, in some situations, may beinconvenient to adjust the exposure. Process 520 of the presentinvention can provide a high dynamic range image by combining multipleimages of the scene taken at various illumination intensity orbrightness levels (instead of or in addition to varying the exposure).The images can be combined and tone-mapped, generating an image that istruer to what is seen by human eyes, which naturally have high dynamicrange.

At steps 522 and 524, an imaging device of the present invention can setthe illumination brightness or intensity level to a first value andcapture an image of the scene at that first illumination level. At steps526 and 528, the imaging device can set the illumination brightness orintensity level to a second value and capture an image of the scene atthat second illumination level.

At step 530, the imaging device can normalize each image to itsrespective illumination level. In one embodiment, the value for eachcolor of each pixel of each captured image can be divided by therelative illumination brightness or intensity level used to capture theimage. For example, if the image captured in step 524 has anillumination level of unity and the age captured in step 528 has anillumination level that is 256 times brighter, then the value of eachcolor of each pixel of the image captured in step 528 can be divided by256.

Thereafter, in step 532, the imaging device can combine the normalizedimages. For example, the imaging device can add the normalized valuesfor each color of corresponding pixels of all of the captured images. Instep 534, the imaging device can tone-map the combined images usingtechniques employed, for example, in computer graphics and photography.

While process 520 shown in FIG. 5B illustrates only two images capturedusing two illumination brightness or intensity levels, an imaging systemof the present invention also can capture more images using moreillumination brightness or intensity levels. For example, an 8-bitimaging device of the present invention can capture three images of ascene using three different illumination brightness or intensity levels:bright illumination (e.g., the unity level), illumination at a quarterof the unity level, and illumination at four times the unity level. Theimages can be normalized and combined, thereby generating an image with4 additional bits-2 bits from each of the two extra exposures. Eachfactor of two in brightness is one additional bite. The combined imagethen can be tone-mapped back into 8 bits of normal contrast range fordisplay and storage.

In an alternative embodiment of the present invention, the imagingdevice also can be configured to adjust the exposure of the camera alongwith the illumination brightness or intensity level to capture multipleimages of the same scene.

In some embodiments, an imaging system may use a particular method tocapture images of scenes that include a wide range of ambient light. Forexample, different types of illumination can be used to generate imagesthat each properly captures a different portion of a scene. These imagescan then be combined to form one image that depicts the entire scene.FIG. 6 shows an illustrative flowchart of method 600 for capturing andcombining images in accordance with one embodiment of the presentinvention.

Like methods 500 and 520, method 600 involves capturing a series ofimages and then combining the images, and the same qualities, such asimage capture speed and illumination adjustment speed, are desirous inan imaging system for the aforementioned reasons.

At step 610, data is collected. Step 610 is similar to step 410 ofmethod 400 and the previous description of step 410 can be applied tostep 610.

At step 620, a scene is divided into portions. In some embodiments, animaging system can analyze a scene to identify regions of the scene withsubstantially different ambient light (e.g., regions where the colorand/or intensity of ambient light differs more than some predeterminedthreshold). It may be advantageous to define the boundaries of theseregions based on the particular scene. For example, system may identifygroups of pixels in a scene that have a similar level of color pixelsaturation (e.g., in red, green, and/or blue) and define that group ofpixels as a region. The system can then record the boundaries of theseregions such that an image of the scene can be divided into speciallydefined portions. Accordingly, an imaging system may divide an imageinto portions that are shaped to fit the particular regions of the scene(e.g., a shape that follows the boundary between two different ambientlight regions). For example, if a person in the foreground of a scene isin a shadow and the background of the scene is brightly lit, an imagingsystem may divide an image of the scene into different portions with theboundary being the outline of the person in the foreground.

In some embodiments, an imaging system can divide an image intopre-defined portions. (e.g., a grid of four or eight portions) andanalyze each portion to determine the amount of ambient light. If animaging system determines that some of the pre-defined portions havesimilar amounts of ambient light, those portions may be grouped togetherto form a larger portion. Using pre-defined portions may be less precisein defining the ambient light regions of a scene but may require lessprocessing power. For example, it may require significantly lessprocessing power to average the color pixel saturation (e.g., in red,green, and/or blue) of each pre-defined portion rather than analyzing anentire image and defining the boundaries of each region.

At step 630, a set of ambient light parameters are generated for eachportion of the image. The ambient light parameters may be generated fromthe collected data. Step 630 can be understood as carrying cut step 420of method 400 for each portion of the image. Accordingly, the previousdescription of step 420 can be applied to the generation of parametersin step 630 with the understanding that each portion of the scene instep 630 corresponds to the entire image in the discussion of step 420.For example, each portion of the image in step 630 may be analyzed asindependent image for the purposes of generating a set of ambient lightparameters.

At step 640, illumination may be provided based on one of the sets ofambient light parameters. At step 650, an image may be captured whileillumination is being provided. Steps 640 and 650 are similar to,respectively, steps 430 and 440. Accordingly, the previous descriptionof steps 430 and 440 can be applied, respectively, to steps 640 and 650.

At step 660, method 600 may diverge depending on how many portions thescene was divided into in step 620 and how many images have beencaptured thus far. If an image has not been captured for each portion,method 600 may proceed with step 640 and the scene may be illuminatedaccording to a new set of parameters. At step 650, a new image can becaptured and this illumination and capturing sequence can repeat untilan image for each portion has been captured. Steps 640, 650 and 660 mayoccur in rapid succession so that the scene does not changesubstantially between each captured image. In some embodiments, animaging system may instruct a user (e.g., using user interface 140) toremain still during image capture.

It is understood that the number of scene portions may be limited sothat the system can reasonably capture an image for each portion in ashort amount of time. For example, an imaging system may limit thenumber of portions that a scene can be divided into at four so that thesystem has time to capture a series of four images without the scenesubstantially changing. Once an image has been captured for eachportion, method 600 may proceed with step 670.

At step 670 the images may be combined. An Imaging system may include aprocessor (e.g., processor 112) or application specific hardware forcombining the images. The processor or application specific hardwarethat combines the images may be located anywhere in the imaging system(e.g., in camera 120 or imaging device 110). In some embodiments, theimages may be combined such that the relevant portion of each image ismerged together to create a combined image that depicts the entirescene. In combining the images, the boundaries of each region can beused to determine which portion of each image should be in the combinedimage. In some embodiments, each image portion might not be croppedprecisely the boundary of a region but fade or feather into theneighboring region so that the combined image does not have any abrupttransitions between the different portions.

The combined image may be stored on the imaging system for late use.Data relating to the image portion boundaries, correspondingillumination for each portion and/or the time difference between theimages may be stored with or appended to the image data. Such data maybe useful when editing or analyzing the image.

It will be understood that the foregoing is only illustrative of theprinciples of the invention, and that various modifications can be madeby those skilled in the art without departing from the scope and spiritof the invention, and the present invention is limited only by theclaims that follow.

What is claimed is:
 1. A method for capturing an image by an imagecapturing device, the method comprising: capturing, with an image sensorof the image capturing device, a first image of a scene illuminated witha first light source at a first illumination level; dividing the firstimage into two or more pre-defined image portions; analyzing the two ormore pre-defined image portions; designating a first portion of adisplay of the image capturing device as an adjustable illuminationsource; providing illumination, by the adjustable illumination source,to illuminate the scene based on results of the analysis of the two ormore pre-defined image portions, with the adjustable illumination sourceat a second illumination level capturing a second image with the imagesensor while illuminating the scene with the adjustable illuminationsource at the second illumination level; combining the first and secondimages to generate a third image; analyzing the third image; andadjusting the illumination of the adjustable illumination source basedon the analysis of the third image.
 2. The method of claim 1, whereinanalyzing the third image comprises analyzing a color balance of thethird image.
 3. The method of claim 1, further comprising: establishing,by the image capturing device, a communication with at least oneexternal device to perform a video-conference; and displaying a fourthimage associated with the video-conference in a second portion of thedisplay of the image capture device wherein the first portion of thedisplay is non-overlapping with the second portion of the display; andcapturing a video of the scene, by the image sensor, using the adjustedillumination to be transmitted in the video-conference.
 4. The method ofclaim 3, further comprising transmitting the third image as a part ofthe video-conference.
 5. The method of claim 1, further comprisingadjusting a brightness level of the first portion of the display.
 6. Themethod of claim 1, further comprising adjusting a color of the firstportion of the display.
 7. The method of claim 1, wherein the two ormore pre-defined image portions comprise less than all of the firstimage.
 8. The method of claim 1, wherein the provided illumination isbased on results of an analysis of less than all of the two or morepre-defined image portions of the first image.
 9. An electronic device,comprising: an image sensor; a display; a memory coupled to the imagesensor; one or more processors coupled to the display, the image sensor,and the memory and configured to execute instructions to cause theelectronic device to: capture, with the image sensor, a first image of ascene illuminated with a first light source at a first illuminationlevel; divide the first image into two or more pre-defined imageportions; analyze the two or more pre-defined image portions; designatea first portion of the display of the electronic device as an adjustableillumination source; provide illumination, by the adjustableillumination source, to illuminate the scene based on results of theanalysis of the two or more pre-defined image portions, with theadjustable illumination source at a second illumination level capture asecond image with the image sensor while illuminating the scene with theadjustable illumination source at the second illumination level; combinethe first and second images to generate a third image; analyze the thirdimage; and adjust the illumination of the adjustable illumination sourcebased on the analysis of the third image.
 10. The electronic device ofclaim 9, wherein the one or more processors is further configured toexecute instructions to cause the electronic device to analyze the thirdimage by executing instruction to cause the electronic device to analyzea color balance of the third image.
 11. The electronic device of claim9, wherein the one or more processors is further configured to executeinstructions to cause the electronic device to: establish, by theelectronic device, a communication with at least one external device toperform a video-conference; and display a fourth image associated withthe video-conference in a second portion of the display of theelectronic device wherein the first portion of the display isnon-overlapping with the second portion of the display; and capture avideo of the scene, by the image sensor, using the adjusted illuminationto be transmitted in the video-conference.
 12. The electronic device ofclaim 11, wherein the one or more processors is further configured toexecute instructions to cause the electronic device to transmit thethird image as a part of the video-conference.
 13. The electronic deviceof claim 9, wherein the one or more processors is further configured toexecute instructions to cause the electronic device to adjust abrightness level of the first portion of the display.
 14. The electronicdevice of claim 9, wherein the one or more processors is furtherconfigured to execute instructions to cause the electronic device toadjust a color of the first portion of the display.
 15. A non-transitoryprogram storage device comprising instructions stored thereon to causeone or more processors to: capture, with an image sensor of an imagecapturing device, a first image of a scene illuminated with a firstlight source at a first illumination level; divide the first image intotwo or more pre-defined image portions; analyze the two or morepre-defined image portions designate a first portion of a display of theimage capturing device as an adjustable illumination source; provideillumination, by the adjustable illumination source, to illuminate thescene based on results of the analysis of the two or more pre-definedimage portions, with the adjustable illumination source at a secondillumination level capture a second image with the image sensor whileilluminating the scene with the adjustable illumination source at thesecond illumination level combine the first and second images togenerate a third image; analyze the third image; and adjust theillumination of the adjustable illumination source based on the analysisof the third image.
 16. The non-transitory program storage device ofclaim 15, wherein the instructions for analyzing the third imagecomprises instruction to cause the one or more processors to analyze acolor balance of the third image.